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FOREWORD 


The  experimental  study  of  the  role  of  feedback  in  computer-based  training  was  conducted 
within  the  independent  research  program  (Program  Element  0601 152N,  Work  Unit  R0001.01) 
under  the  sponsorship  of  the  Chief  of  Naval  Research.  The  goal  of  this  study  was  to  determine  the 
relative  effectiveness  ot  different  types  of  feedback  in  learning  how  to  operate  a  program  entry 
panel  using  computer-based  training.  The  success  of  future  Navy  computer-based  teaching  systems 
hinges  on  effective  employment  of  feedback  during  computer-based  instruction.  This  report 
provides  preliminary  results  on  the  benefits  of  different  types  of  feedback  employed  during 
computer-based  training. 


T.  F.  FINLEY  RICHARD  C.  SORENSON 

Captain,  U.S.  Navy  Technical  Director  (Acting) 

Commanding  Officer 
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SUMMARY 


Problem 

Navy  personnel  often  have  difficulty  operating  the  state-of-the-art  programmable  equipment 
employed  in  iadar  systems,  communication  systems,  and  transportation  systems.  These  types  of 
devices  are  often  designed  without  adequate  consideration  of  the  user  interface.  Computer-based 
training  (CBT)  systems  have  been  developed  to  help  users  overcome  difficulties  associated  with 
learning  to  operate  complex  devices.  An  important  capability  of  CBT  is  feedback  that  informs 
users  about  the  correctness  of  their  knowledge  of  device  procedures.  Current  research  in  CBT 
provides  little  guidance  as  to  when  feedback  should  be  provided  and  how  to  design  feedback 
content. 

Objective 

The  objective  of  this  study  was  to  determine  the  relative  effectiveness  of  different  types  of 
feedback  in  learning  how  to  operate  a  program  entry  panel  during  computer-based  training. 

Approach 

An  experimental  CBT  lesson  on  how  to  operate  a  military  phone  system  was  administered  to 
80  Navy  students.  The  lesson  was  presented  individually  on  a  microcomputer  and  consisted  of  an 
introduction,  a  practice,  and  a  performance  test.  During  practice,  each  treatment  group  received 
one  of  four  types  of  feedback.  The  computer  provided  feedback  either  immediately  following  an 
error  or  at  the  end  of  the  button-pushing  of  the  to-be-leamed  sequence.  Feedback  consisted  of  the 
correct  response  or  a  “wrong”  indication. 

Results 

All  the  CBT  treatment  groups  outperformed  a  no-treatment  control  group.  The  treatment  group 
who  received  delayed  feedback  performed  significantly  better  on  the  performance  test  than  those 
who  received  immediate  feedback.  Delaying  the  feedback  was  beneficial  during  CBT  because  it 
aids  in  the  development  of  a  usable  device  schema. 
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INTRODUCTION 


The  Problem 

Navy  personnel  often  have  difficulty  operating  the  state-of-thc  -art  programmable  equipment 
employed  in  radar  systems,  communication  systems,  and  transportation  systems.  These  complex 
devices  “often  contain  internal  mechanisms  that  obscure  the  relationship  between  the  user’s  input 
and  the  device’s  behavior”  (Shrager  &  Klahr,  1986,  p.  153).  These  types  of  devices  tend  to  be 
designed  without  adequate  consideration  of  the  user  interface.  Indeed,  conceptual  models  of  how 
the  device  works  are  used  in  the  engineering  of  the  device  (Norman,  1988),  but  engineers  often 
seem  to  give  little  thought  on  how  the  user  will  make  it  work  (e.g.,  pushing  buttons,  flipping 
switches). 

Computer-based  training  (CBT)  systems  have  been  developed  to  help  users  overcome  the 
learning  difficulties  associated  with  operating  these  types  of  complex  devices.  CBT  systems  are 
used  extensively  at  many  industrial  and  military  installations.  Computers  are  used  to  teach 
operation  of  the  device  because  they  provide  users  with  a  safe  environment  in  which  to  learn  how 
to  operate  equipment  without  endangering  themselves  and  others  or  harming  the  equipment.  Also, 
CBT  systems  can  be  easily  programmed  to  simulate  faults.  Consequently,  CBT  is  more  flexible 
than  using  the  actual  equipment  for  training. 

Capabilities  of  CBT  systems  include  the  use  of  sound,  animated  graphics,  and  video  and  the 
presentation  of  information  in  a  linear  or  interactive  mode.  Feedback  for  student  responses  is  also 
an  important  capability  of  CBT.  In  order  for  CBT  to  be  effective,  it  must  use  feedback  strategies 
that  allow  the  student  to  learn  device  procedures.  Feedback  is  important  for  learning  how  to  operate 
a  complex  device,  in  that  it  informs  the  student  about  the  correctness  of  his  or  her  knowledge  of 
the  device  procedures. 

Instructional  designers  implement  feedback  in  current  CBT  systems  in  many  ways.  Feedback 
is  typically  provided  immediately  after  a  student  responds  to  an  inquiry  from  the  training  system, 
but  feedback  has  also  been  provided  at  some  later  point  during  the  instruction.  For  example,  some 
CBT  systems  such  as  the  H-53  helicopter  training  system  (Cowen,  1989)  provide  feedback  only 
after  three  errors  in  the  practice  of  a  single  subtask.  In  contrast,  the  E-2C  radar  training  system 
(Stem  &  McCabe,  1987)  provides  feedback  immediately  after  the  completion  of  every  subtask. 
Another  training  system,  the  ECM/ECCM  radar  jamming  training  system  (Cowen,  1985),  provides 
feedback  only  after  the  completion  of  the  entire  radar  jamming  procedure. 

The  effectiveness  of  feedback  hinges  on  the  suitability  of  its  content  to  the  learning  task.  The 
two  most  common  types  of  feedback  used  in  CBT  are  confirmatory  feedback  and  corrective 
feedback.  Confirmatory  feedback  provides  an  indication  of  correctness  of  input  such  as  the 
presentation  “right”  or  “wrong”  as  the  computer’s  response  to  the  student’s  input.  Corrective 
feedback  presents  the  correct  answer  for  the  student’s  incorrect  response.  For  example,  if  the 
student  answered  “gas”  to  the  question,  “What  liquid  do  you  put  in  the  radiator?”  the  computer 
would  respond,  “The  radiator  should  be  filled  with  water.” 

Although  meta-analyses  have  demonstrated  that  computerized  instruction  can  improve  student 
achievement  (Bangert-Drowns,  C.  C.  Kulik,  &  J.  A.  Kulik,  1985;  J.  A.  Kulik,  &  C.  C.  Kulik,  1988; 
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Kulik,  Kulik,  &  Bangert-Drowns,  1985;  Niemiec  &  Walberg,  1987;  Roblyer,  1990),  there  is  no 
clear  indication  in  the  literature  as  to  when  feedback  should  be  provided  during  a  CBT  lesson  and 
what  its  content  should  be.  An  important  question  is  how  to  use  feedback  most  effectively  in  a 
computerized  lesson  on  how  to  operate  the  program  entry  panel  (PEP)  for  a  complex  device. 

The  purpose  of  this  study  was  to  investigate  the  relative  effectiveness  of  different  types  of 
feedback  in  learning  how  to  operate  a  PEP  device  using  CBT. 

Background 

Operating  a  Program  Entry  Panel  (PEP)  Device 

The  knowledge  of  how  to  operate  a  PEP  device  has  been  described  as  a  synthetic  construction 
of  familiar  items  (e.g.,  buttons,  switches)  in  a  novel  fashion  (Moran,  1981).  The  student’s 
knowledge  of  how  to  perform  a  device’s  unique  sequence  of  operations  is  called  a  task/action 
mapping  (Young,  1981).  Task/action  mappings  have  been  used  to  describe  the  operation  of  the 
standard  four-function  calculator  (Young,  1983).  This  representation  accurately  describes  not  only 
the  procedures  for  basic  calculations  such  as  2  +  3  =  5,  but  also  for  special  calculations  such  as 
cumulative  calculations,  constant  calculations,  and  upside-down  division.  For  instance,  the 
quotient  8  /  (4  +  6)  can  only  be  evaluated  by  pushing  the  button  sequence,  4,  +,  6,  -5-,  =,  8,  =.  The 
user’s  task/action  mapping  of  the  device  helps  the  user  retrieve  the  sequences  of  push  buttons 
operation  needed  to  perform  certain  types  of  calculations. 

Shrager  and  Klahr  (1986)  called  this  representation  a  device  schema  and  they  proposed  three 
subclasses  of  knowledge  that  characterize  how  a  device  works:  syntactic  knowledge,  semantic 
knowledge,  and  device  model  knowledge.  Syntactic  knowledge  is  the  grammar  of  the  sequence. 
The  user  must  understand  what  constitutes  a  “legal”  sequence  of  buttons  and  switches.  For 
instance,  to  compute  a  basic  calculation  on  the  four-function  calculator,  a  user  must  always  push  a 
number  button  followed  by  an  operation  button,  followed  by  a  number  button,  followed  by  an 
operation  button. 

Semantic  knowledge  is  knowing  the  results  of  pushing  an  individual  button  or  flipping  a 
particular  switch.  For  example,  when  performing  a  basic  calculation  on  the  four-function 
calculator,  the  meaning  of  the  5  button  is  the  integer  Five  and  the  meaning  of  the  =  button  is  equals. 
The  function  of  a  button,  however,  may  change  with  the  type  of  calculation  to  be  performed. 
During  a  cumulative  calculation  the  =  may  also  mean  addition  as  in  the  sequence  4,  +,  6,  =,  =. 
When  a  button  or  a  series  of  buttons  represents  more  than  one  function,  the  user  may  become 
confused  and  consequently  will  have  difficulty  operating  the  device. 

Device  model  knowledge  is  the  user’s  understanding  of  the  interrelationships  among  the 
internal  mechanisms  of  the  device.  For  example,  a  user’s  device  model  of  a  four-function 
calculator  consists  of  a  vague  notion  that  pushing  number  buttons  puts  integers  into  number 
registers  inside  the  calculator  and  that  pushing  operation  buttons  affects  the  contents  of  these 
internal  registers  and  of  the  numbers  that  appear  in  the  external  display.  A  user’s  device  model  can 
mediate  the  recall  of  syntactic  and  semantic  knowledge.  For  instance,  the  user  executes  his  or  her 
device  model  of  the  four-function  calculator  and  knows  that  in  order  to  enter  a  number  into  an 
internal  register  of  the  calculator  he  or  she  must  push  an  operation  button  after  entering  the 
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numerals  of  the  first  operand.  Thus,  the  device  model  of  how  the  user  thinks  the  device  works  has 
helped  in  remembering  the  syntax  of  how  to  perform  a  basic  calculation.  Shrager  and  Klahr  (1986) 
report  that  some  subjects  in  their  experiment  were  successfully  able  to  operate  the  “BigTrak” 
programmable  toy  tank  using  a  device  schema  of  how  the  toy  works. 

Using  Computers  to  Teach  Device  Operation 

CBT  “generally  refers  to  the  use  of  computers  to  accomplish  a  specific  training  objective” 
(Willis,  1987,  p.  80).  There  are  CBT  systems  at  industrial  and  military  installations  that  teach 
operation  of  complex  physical  devices  such  as  radar  equipment,  communication  systems,  and 
military  vehicles  (Cowen,  1985;  Malec  &  Luszczak,  1987;  Stem  &  McCabe,  1987). 

Computerized  instruction  is  a  convenient  way  to  present  consistent  information  to  students  at 
different  training  sites.  Other  advantages  of  computerized  instruction  include  standardizing  course 
materials,  matching  materials  to  the  ability  of  the  student,  and  automated  scoring  of  performance 
tests.  Computers  also  provide  a  safe  environment  in  which  to  operate  potentially  dangerous 
equipment.  Procedures  can  be  attempted  which,  if  done  incorrectly,  would  endanger  the  student  or 
the  instructor  and  harm  the  equipment.  :or  example,  CBT  could  provide  a  student  with 
opportunities  to  practice  the  adjustment  of  temperature  settings  during  a  power  plant  operation 
lesson.  Coleman  (1988)  reported  student  achievement  using  CBT  equals  that  achieved  using  the 
actual  complex  device.  As  a  result,  CBT  is  cost  effective  because  it  typically  costs  much  less  than 
the  actual  equipment.  Coleman  also  reported  that  CBT  “consistently  took  less  time  to  deliver 
course  material  than  the  same  course  taught  conventionally,  generally  30  per  cent  less  time”  (p. 
44). 


Feedback  in  Computer-based  Training 

Feedback  for  student  responses  is  a  key  design  feature  of  CBT  systems.  Most  researchers  and 
practitioners  agree  that  feedback  is  an  important  factor  in  learning  (Annett,  1969;  Gagne,  1977; 
Kulhavy,  1977).  A  meta-analysis  of  15  studies  (Schimmel,  1983)  indicates  that  using  feedback  in 
computer-based  instruction  is  more  effective  than  not  using  feedback.  Schimmel  located  15  studies 
that  investigated  both  feedback  and  no-feedback  conditions  in  administering  an  adult  lesson 
involving  meaningful  verbal  material.  Schimmel  reported  that  subjects  receiving  feedback  on 
average  scored  .47  standard  deviations  higher  than  those  receiving  no  feedback.  However,  this 
analysis  did  not  show  how  the  timing  and  content  of  feedback  in  computer-based  instruction  may 
be  related  to  learning  and  retention  of  the  device  procedures. 

Feedback  For  Correct  Responses 

The  purpose  of  providing  feedback  for  correct  answers  is  to  help  the  student  perceive  that  his 
or  her  understanding  of  the  subject  matter  is  correct.  Kulhavy  and  others  (Kulhavy  &  Anderson, 
1972;  Kulhavy  &  Parsons,  1972;  Kulhavy  &  Swenson,  1975)  found  that  correct  responses  were 
repeated  whether  or  not  feedback  was  provided.  These  findings  are  consistent  with  the  results  of 
earlier  studies  (Krumboltz  &  Kiesler,  1965;  Oppenheim,  1964)  which  investigated  feedback  for 
correct  responses  during  verbal  learning.  These  studies  found  that  confirmation  did  not  affect 
student  achievement.  Steinberg  (1984)  believed  that  there  is  no  need  to  provide  extensive  feedback 
after  a  correct  response:  “It  serves  no  useful  purpose,  and  the  students  will  not  read  it”  (p.  86). 
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In  contrast,  feedback  for  incorrect  responses  is  highly  important  to  learning.  Buss  (Buss  & 
Buss,  1956;  Euss,  Weiner,  &  Buss,  1954)  found  that  subjects  who  were  informed  only  of  their 
errors  performed  better  than  subjects  who  received  feedback  only  for  correct  responses.  Although 
feedback  for  correct  responses  can  be  helpful  in  some  settings,  this  investigation  focused  on  the 
effects  of  providing  feedback  for  student  errors.  The  effectiveness  of  feedback  for  student  errors 
during  CBT  may  be  related  to  when  it  is  provided  and  the  suitability  of  its  content. 

Timing  of  Feedback 

A  literature  review  of  the  results  comparing  immediate  to  delayed  feedback  in  the  education 
literature  (Wager  &  Wager,  1985)  indicates  that  both  types  of  feedback  are  effective  in  promoting 
learning.  The  feedback  content  in  most  of  these  studies  was  confirmatory  and  the  feedback  delay 
was  tvpically  a  day  after  the  lesson  examination.  For  example,  Robin  (1978)  found  no  differences 
in  achievement  or  study  time  between  immediate  and  delayed  feedback  conditions  during  a  self- 
paced  sectior  of  a  college-level  abnormal  psychology  course.  The  feedback  in  this  study  was  a 
grade  of  “correct”  or  “incorrect”  by  the  class  proctor,  and  the  proctor  did  not  provide  correct 
answers  for  incorrect  responses.  During  the  delayed  condition,  this  feedback  was  not  provided  until 
the  beginning  of  the  next  class  period. 

A  recent  meta- analysis  by  (J.  A.  Kulik  &  C.  C.  Kulik,  1988)  of  53  studies  on  feedback  timing 
and  verbal  learning  also  found  mixed  results.  The  analysis  indicated  that  immediate  feedback  was 
more  effective  than  delayed  feedback  for  applied  studies  using  actual  classroom  quizzes.  However, 
they  also  found  that  delayed  feedback  was  superior  to  immediate  feedback  for  experimental  studies 
involving  the  learning  of  multiples-choice  items  or  lists  of  words.  It  is  difficult  to  draw  conclusions 
from  this  study  because  many  of  the  journal  articles  cited  in  the  meta-analysis  sometimes  classify 
after-test  feedback  as  immediate  feedback  and  sometimes  as  delayed  feedback.  The  inconsistent 
use  of  the  terms  immediate  and  delayed  has  made  it  difficult  to  generalize  research  results  on 
feedback  (Dempsey  &  Wager,  1988). 

Nevertheless,  there  is  strong  evidence  that,  at  least  for  some  paradigms,  delayed  feedback  for 
student  errors  is  more  effective  than  immediate  feedback.  Delayed  feedback  groups  consistently 
outperformed  no-delay  feedback  groups  in  paired-associate  tasks,  lessons  with  multiple-choice 
items,  and  learning  prose  passages  (Kulhavy,  1977).  Kulhavy  and  Anderson  (1972)  argued  that 
delayed  feedback  is  more  effective  because  an  interval  between  response  and  feedback  reduces  the 
strength  of  the  memory  trace  for  the  incorrect  response.  They  argued  that,  when  feedback  is 
provided  immediately,  the  wrong  response  will  interfere  with  the  correct  response  (i.e.,  pro-active 
interference).  They  found  that  subjects  spend  significantly  more  time  attending  to  delayed 
feedback.  Kulhavy  and  Parsons  (1972)  found  that  delayed  feedback  was  most  effective  when  the 
level  of  the  instructional  material  was  appropriate  for  the  student  and  the  student  could  relate  the 
material  to  some  previously  acquired  knowledge. 

Content  of  Feedback 

As  previously  mentioned,  a  meta-analysis  was  performed  by  Schimmel  (1983)  on  the 
educational  effectiveness  of  feedback  in  computerized  instruction.  Although  feedback  treatment 
groups  generally  outperformed  their  respective  control  groups,  this  study  found  no  significant 
advantage  for  one  feedback  content  type  over  another.  For  example,  the  effect  size  for  a  study 
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(Kulhavy,  Yekovich,  &  Dyer,  1976)  which  administered  review  type  feedback,  was  .71  while  the 
effect  size  for  a  study  (Roper,  1977)  that  provided  “right/wrong”  feedback  was  .69.  Moreover, 
studies  providing  the  correct  response  as  feedback  yielded  effect  sizes  ranging  from  -1.25  to  1.10. 
Schimmel  (1988)  concluded  that  the  empirical  research  on  the  content  of  feedback  provides  little 
guidance:  “Providing  learners  with  extensive  information  about  why  their  answers  arc  correct  or 
incorrect  has  proven,  on  the  whole,  no  more  useful  than  offering  minimal  information”  (p.  186). 
Anderson,  Kulhavy,  and  Andre  (1971)  found  similar  results  when  college  students  were  required 
to  learn  about  heart  disease  using  a  1 12-frame  computer-based  lesson.  The  college  students  were 
provided  feedback  either  after  right  answers  or  wrong  answers  but  not  both.  Result  showed  that, 
although  both  groups  committed  about  the  same  number  of  errors  during  the  lesson,  the  wrong- 
only  condition  had  higher  performance  test  scores.  Kulhavy  (1977),  after  reviewing  the  literature 
on  feedback  in  written  instruction,  concluded  that  feedback  works  best  when  its  content  “identifies 
and  corrects  errors-or  allows  the  learner  to  correct  them”  (p.  229).  However,  Kulhavy  noted  that 
this  content  is  only  effective  if  the  correct  answer  is  not  blatantly  available  before  the  student 
response  (so  it  will  not  be  copied)  and  only  if  the  lesson  subject  matter  can  be  comprehended  by 
the  student. 

The  research  suggests  that  instruction  can  be  effective  with  feedback  that  is  confirmatory  or  the 
corrective,  and  immediate  or  delayed.  All  the  while,  there  is  some  indication  that  delayed  feedback 
may  be  may  be  slightly  more  effective  than  immediate  feedback.  However,  most  of  the  research 
studied  feedback  in  the  learning  of  factual  or  declarative  knowledge  and  not  procedural  knowledge. 
These  studies  examined  feedback  in  instruction  to  perform  tasks  that  are  different  than  operating 
the  PEP  devices  which  are  of  interest  here.  In  fact,  no  research  exists  on  the  role  of  feedback  in 
learning  how  to  operate  PEP  devices  such  as  automated  bank  tellers,  telecommunications 
equipment,  or  avionics  consoles. 

The  only  research  which  investigated  the  role  of  feedback  in  learning  similar  procedures  was 
done  by  Lewis  and  Anderson  (1985).  They  compared  the  effectiveness  of  immediate  and  delayed 
feedback  in  learning  the  rules  in  a  computerized  adventure  game.  In  this  experiment,  subjects 
would  attempt  to  move  from  room  to  room  in  search  of  treasures  by  activating  objects  in  each  room 
(e.g.,  waving  a  wand,  rubbing  the  lamp).  In  the  delayed  feedback  condition,  subjects  were  allowed 
to  move  into  the  next  room  by  activating  the  wrong  object  but  could  not  move  into  a  third  room 
until  the  correct  object  was  activated  in  the  first  room.  In  the  immediate  feedback  condition, 
subjects  were  informed  immediately  of  an  error.  Results  showed  that  the  immediate  feedback 
ubjects  performed  more  accurately  during  the  performance  test  than  the  delayed  feedback 
subjects.  However,  the  subjects  in  the  delayed  feedback  group  tested  more  hypotheses  and  were 
better  at  recognizing  dead  ends.  The  authors  of  the  study  concede  that  “subjects  in  the  immediate 
feedback  condition,  although  significantly  more  accurate  at  selecting  forward  moves,  might 
flounder  if  allowed  to  move  off  of  the  correct  solution  path”  (p.  63).  In  short,  the  issue  of  immediate 
versus  delayed  feedback  is  “controversial”  (Anderson,  1987). 

Hypotheses 

The  key  dimensions  of  CBT  feedback  to  student  errors  are  content  and  timing.  The  content  of 
feedback  in  CBT  is  typically  either  an  indication  of  right  or  wrong  (i.e.,  confirmatory  feedback)  or 
the  presentation  of  the  correct  response  (i.e.,  corrective  feedback).  It  was  hypothesized  that  (1) 
feedback  in  CBT  for  operating  a  PEP  device  is  more  effective  when  it  is  confirmatory  than  when 
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it  is  corrective  and  (2)  feedback  in  CBT  for  operating  a  PEP  device  is  more  effective  when  it  is 
delayed  than  when  it  is  immediate.  Users  successfully  operate  PEP  devices  by  retrieving  their 
mental  representation  (i.e.,  device  schema)  of  how  to  make  the  device  work.  They  construct  a 
device  schema  by  formulating  hypotheses  about  how  to  make  the  device  work  and  then  by  testing 
those  hypotheses.  During  CBT,  subjects  can  test  hypotheses  by  interacting  with  a  simulation  of  the 
device.  Feedback  during  CBT  is  most  important  when  it  assists  the  user  in  confirming  or  rejecting 
hypotheses.  When  proposed  hypotheses  are  wrong,  the  user  will  search  the  problem  space  and 
formulate  and  test  new  hypotheses.  Accordingly,  feedback  in  the  form  of  a  “you  are  wrong” 
statement  in  response  to  a  student  error  (when  compared  to  feedback  in  the  form  of  the  correct 
response)  leads  to  active  hypothesis  testing  which  produces  a  rich  device  schema  which  leads  to 
better  learning.  Lewis  and  Anderson  (1985)  found  that  subjects  who  actively  tested  hypotheses 
were  better  performers.  Users  are  unlikely  to  formulate  and  test  new  hypotheses  if  provided  with 
the  correct  response. 

During  CBT,  feedback  can  be  provided  immediately  after  an  error  or  at  the  end  of  the  practice 
for  a  specific  task.  Delaying  the  feedback  until  after  the  end  of  practice  for  each  task  will  allow  the 
student  to  formulate  and  refine  hypotheses.  Users  who  formulate  and  test  hypotheses  in  the 
learning  of  a  PEP  procedure  will  generate  device  schema  that  will  help  them  recall  device 
procedures.  Consequently,  it  was  predicted  that  (1)  subjects  provided  with  confirmatory  (e.g.,  “you 
are  wrong”)  feedback  for  errors  they  made  during  practice  of  a  to-be-learned  sequence  will  leam 
how  to  perform  more  device  procedures  than  those  provided  with  corrective  feedback  and  (2) 
subjects  provided  with  feedback  at  the  end  of  practice  of  a  to-be-learned  sequence  will  leam  how 
to  perform  more  device  procedures  than  those  provided  with  feedback  immediately  after  every 
error  during  practice  of  a  to-be-learned  sequence. 

In  short,  it  was  predicted  that  subjects  provided  with  confirmatory  or  delayed  feedback  during 
CBT  will  outperform  those  provided  with  corrective  or  immediate  feedback.  The  users  provided 
with  confirmatory  or  delayed  feedback  will  have  a  better  understanding  of  the  relationship  among 
device  buttons  and  will  perform  better  on  a  performance  test  administered  immediately  following 
the  CBT. 


METHOD 


Subjects 

The  subjects  in  this  study  were  53  Navy  students  awaiting  instruction  for  Sonar  Technician 
(ST)  “A”  school  at  the  Fleet  Anti-submarine  Warfare  (ASW)  Training  Center,  Pacific  and  27  Navy 
students  awaiting  instruction  for  Apprenticeship  Training  at  the  Recruit  Training  Command  (RTC) 
in  San  Diego.  More  than  one  sample  was  tested  because  the  number  of  students  at  the  ST  “A” 
school  available  for  testing  was  small.  The  students  enrolled  at  the  ASW  Training  Center 
(henceforth,  referred  to  as  the  ASW  sample)  generally  score  between  the  65th  and  99th  percentile 
on  the  Armed  Forces  Qualification  Test  (Monzon  &  Foley,  1988).  The  mean  enlisted  rank  of  the 
ASW  sample  was  E-3  (rank  ranges  from  E-l  to  E-9)  and  the  mean  age  was  about  21  years.  The 
ASW  subjects  had  been  in  the  Navy  approximately  1 1  months  and  at  the  ASW  training  center 
about  3.5  months. 
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Students  enrolled  at  RTC  Apprenticeship  Training  school  (henceforth,  referred  to  as  the  RTC 
sample)  generally  score  between  the  10th  and  30th  percentile  on  the  Armed  Forces  Qualification 
Test  (Monzon  &  Foley,  1988).  The  mean  enlisted  rank  of  the  RTC  sample  was  E-2  and  the  mean 
age  was  about  21  years.  These  subjects  had  been  in  the  Navy  approximately  7.5  months  and  at  the 
RTC  about  3.5  months. 

Procedure 

The  study  consisted  of  three  phases.  First  the  subjects  were  required  to  fill  out  the  pre¬ 
questionnaire,  which  included  consent  forms.  Next,  they  were  administered  the  experimental  CBT 
lesson.  Lastly,  they  were  required  to  fill  out  the  post-questionnaire.  All  phases  were  administered 
individually  and  were  self-paced. 

Pre-questionnaire 

The  pre-questionnaire  (see  Appendix  A)  asked  about  the  subject’s  mood,  experience  with 
telephones,  microcomputers,  and  video  games  and  for  biodemographic  information.  In  addition, 
each  subject  was  required  to  sign  a  consent  form  to  act  as  a  subject  for  scientific  research.  The 
consent  form  described  the  to-be-performed  task  and  the  objective  of  the  study. 

Computer-based  Training 

Each  subject  was  seated  in  front  of  an  Apple  Macintosh  Model  SE/30  microcomputer.1  The 
microcomputer  presented  a  CBT  lesson  on  how  to  operate  the  Consolidated  Area  Telephone 
System  (CATS).  The  objective  of  the  lesson  was  to  learn  how  to  activate  and  employ  eight  features 
of  the  Navy’s  CATS  using  the  AT&T  Model  2500  Telephone  (AT&T  Technologies  Inc.,  1987). 
The  nucleus  of  the  lesson  consisted  of  a  computerized  graphic  simulation  of  the  Model  2500 
Telephone  (see  Figure  1).  The  simulation  was  displayed  on  the  9-inch  (diagonal)  monochrome 
monitor  of  the  SE/30.  The  subjects  interacted  with  the  simulation  by  “clicking”  a  mouse  on  the 
computerized  graphics,  listening  to  simulated  device  sounds,  and  observing  changes  to  the 
simulation.  The  remaining  lesson  materials  consisted  of  computer  frames  of  text  and  graphics 
which  were  linked  to  the  simulation.  The  lesson  had  three  parts:  introduction,  practice,  and 
performance  test. 

Introduction.  The  introduction  (see  Appendix  B)  presented  frames  of  information  on  how  to 
use  the  mouse,  the  objective  of  the  CBT  lesson,  the  locations  of  the  Model  2500  Telephone  buttons 
(i.e.,  click  areas),  and  a  sample  practice  item.  The  introduction  did  not  present  any  information  on 
how  to  use  the  phone.  To  start  the  practice,  the  subjects  entered  the  last  four  digits  of  their  social 
security  number  (see  Figure  2). 

Practice.  Practice  consisted  of  activating  each  of  the  features  of  CATS  by  clicking  the  mouse 
on  16  active  buttons  or  click  areas  located  on  the  computerized  graphic  representation  of  the  CATS 
phone  (see  Figure  1).  For  example,  if  instructed  to  make  a  call  on  the  CATS  phone,  the  subject 
would  click  the  mouse  on  the  Lift  Receiver  button,  then  on  the  Listen  For  Tone  button,  and  then 
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Figure  1.  Screen  of  AT&T  Model  2500  Telephone  and  example  of  task. 


a. SB 


Presentation  Window 


BEFORE  WE  START,  PLEASE  CLICK 
THE  LAST  FOUR  DIGITS  OF  YOUR 
SOCIAL  SECURITY  NUMBER  USING 
THE  KEYPAD  TO  THE  LEFT. 

IF  YOU  HAKE  AN  ERROR, 
PLEASE  START  OVER  BY 
CLICKING  IN  THE  BOX  BELOW. 


To  Start  Over, 
Click  On  This  Box 


(Click  On 
Numbers, 
Please) 


Figure  2.  Sample  screen  of  entering  identification  for  practice. 


seven  times  on  the  number  buttons.  If  the  subject  made  an  error  during  practice,  the  CBT  system 
provided  feedback.  The  type  of  feedback  provided  will  be  described  later. 

Before  the  practice  of  each  task,  the  subject  was  provided  with  a  general  description  of  the 
purpose  of  the  task  (Figure  3),  but  not  how  to  do  it.  During  practice,  the  subject  could  access  the 
information  presented  during  the  introduction  by  clicking  with  a  mouse  on  a  help  button.  The 
practice  contained  eight  tasks  presented  in  the  following  order:  Program  Call  Forwarding,  Cancel 
Call  Forwarding,  Call  Pickup,  Program  Abbreviated  Dialing,  Use  Abbreviated  Dialing,  Call 
Transfer,  Conference  Calling,  and  Call  Hold.  Table  1  lists  the  number  of  steps  required  to  activate 
and  use  each  of  these  tasks  on  the  simulated  CATS  system. 


Presentation  Window 


Call  Forwarding 

Temporarily  fowards  all  your  calls  to  another  extension 
or  to  a  local  outside  number.  Use  when  you  will  be  away 
from  your  phone  and  you  want  your  calls  to  be  forwarded 
to  a  phone  of  your  choice. 


•  This  task  begins  with  the  receiver  in  its  cradle. 


I  TO  CONTINUE,  CLICK  ON  THIS  BOX 


Return 


Figure  3.  Sample  screen  of  description  of  practice  task. 
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Table  1 


Steps  Required  To  Operate  Consolidated  Area  Telephone  System  (CATS)  Features 


Feature  Number  of  Steps  Number  of  Clicks  of  Mouse 


Call  Forwarding,  Program 

7 

13 

Call  Forwarding,  Cancel 

5 

7 

Call  Pickup 

3 

5 

Abbreviated  Dialing,  Program 

11 

20 

Abbreviated  Dialing,  Use 

4 

7 

Call  Transfer 

4 

8 

Conference  Calling 

7 

13 

Call  Hold 

10 

18 

Note.  Listed  in  order  of  presentation  in  computer-based  training. 

Call  Forwarding  sends  all  of  the  user’s  calls  to  another  extension  or  to  a  local  outside  number. 
Call  Forwarding  is  activated  by  entering  a  program  code  (e.g.,  *12)  followed  by  the  extension  or 
outside  telephone  number.  Program  Call  Forwarding  requires  the  user  to  enter  7  steps  (e.g..  Lift 
Receiver,  Enter  *12)  or  13  clicks  on  the  simulated  CATS  system.  Cancel  Call  Forwarding  requires 
5  steps  or  7  clicks. 

Call  Pickup  allows  the  user  to  answer  another  phone  on  his  or  her  phone.  It  allows  the  user  to 
handle  somebody  else’s  calls  without  having  to  leave  his  or  her  space.  Call  Pickup  is  activated  by 
entering  a  program  code  and  requires  3  steps  or  5  clicks. 

Program  Abbreviated  Dialing  creates  a  personal  list  of  phone  numbers  which  can  be  accessed 
by  dialing  only  two  numbers.  It  is  used  to  save  time  for  dialing  frequently  called,  lengthy,  or 
important  numbers.  Both  Program  and  Use  Abbreviated  Dialing  are  activated  by  entering  several 
program  codes  and/or  list  numbers.  Program  Abbreviated  Dialing  requires  1 1  steps  or  20  clicks; 
Use  Abbreviated  Dialing,  4  steps  or  7  clicks. 

Call  Transfer  moves  a  call  from  the  user’s  phone  to  another  extension  or  a  local  outside 
number.  It  is  used  mainly  when  the  caller  needs  to  speak  with  someone  else.  Call  Transfer  is 
activated  by  pushing  the  switch-hook  and  entering  another  number  (4  steps  or  8  clicks). 

Conference  Calling  adds  a  third  party  to  a  call  and  results  in  a  three-way  conversation.  It  is 
activated  by  pushing  the  switch-hook,  entering  another  number,  and  pushing  the  switch-hook  again 
(7  steps  or  13  clicks). 
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Call  Hold  temporarily  disconnects  a  call  so  that  the  user  can  place  another  call  or  activate 
another  feature.  The  Call  Hold  task  requires  the  user  to  activate  Call  Hold,  dial  another  number, 
and  deactivate  Call  Hold  (10  steps  or  18  clicks). 

CATS  represents  a  complex  PEP  device  since  it  has  multiple  features.  Some  CATS  features 
require  simple  procedures  for  correct  operation  while  other  CATS  features  require  complex 
procedures.  The  more  complex  features  require  the  user  to  perform  multiple  functions  and  enter 
many  operational  steps. 

Performance  Test.  The  performance  test  consisted  of  performing  the  eight  tasks  in  the 
practice.  There  was  only  one  performance  item  per  task.  Before  each  item,  as  in  the  practice,  the 
subject  was  provided  with  a  general  description  of  the  task’s  purpose  but  no  information  on  how 
to  do  it.  Each  performance  item  required  the  subject  to  click  the  buttons  in  the  correct  sequence. 
No  feedback  was  provided.  The  performance  tasks  required  the  same  steps  as  the  practice  tasks, 
but  the  extension  numbers  were  different. 

Post-questionnaire 

The  post-questionnaire  (see  Appendix  C)  consisted  of  questions  about  the  subject’s  mood, 
understanding  of  CATS’s  procedures,  and  use  of  the  experimental  CBT.  In  brief,  subjects  were 
asked  which  of  CATS  features  they  felt  comfortable  using  and  to  describe  the  purpose  of  various 
CATS  elements  such  as  the  dial  tone  and  the  switch-hook.  In  addition,  they  were  asked  to  rate  the 
friendliness  of  the  CBT  and  to  comment  on  the  feedback  provided  during  CBT.  They  were  also 
asked  whether  or  not  they  would  recommend  using  CBT  to  teach  control  panel  procedures  and  for 
any  comments. 

Experimental  Design 

Subjects  were  randomly  assigned  to  the  control  group  or  to  one  of  four  treatment  groups.  The 
control  subjects  were  administered  the  pre-questionnaire,  the  CBT  lesson  introduction,  the  CBT 
performance  test,  and  the  post-questionnaire.  They  were  not  provided  with  CBT  practice. 

The  four  treatment  groups  were  provided  all  phases  of  the  CBT.  During  CBT  practice,  each 
group  received  one  of  four  types  of  feedback:  Immediate  confirmatory  feedback,  immediate 
corrective  feedback,  delayed  confirmatory  feedback,  and  delayed  corrective  feedback. 

Immediate  feedback  was  feedback  provided  the  instant  the  subject  made  an  error.  In  the 
immediate  confirmatory  feedback  condition,  feedback  for  an  error  consisted  of  a  a  flashing  box 
containing  the  words  “Incorrect...  Try  Again.”  An  example  of  this  presentation  is  shown  in 
Figure  4. 

In  the  immediate  corrective  feedback  condition,  feedback  for  an  error  contained  the  correct 
response.  This  presentation  (see  Figure  5)  consisted  of  a  hand  pointing  to  the  correct  response  and 
a  flashing  box  containing  the  words  “Incorrect.... The  Hand  Points  to  the  Correct  Response.  (Click 
on  the  Hand).”  In  these  two  conditions,  the  subjects  had  to  repeat  missed  steps  until  they  performed 
them  correctly.  After  all  the  correct  steps  had  been  entered  for  that  task,  subjects  received  the 
message  “You  Have  Correctly  Entered  the  Steps  For....”  and  proceeded  to  the  next  task. 
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Figure  4.  Sample  screen  of  immediate  confirmatory  feedback. 


Figure  5.  Sample  screen  of  immediate  corrective  feedback. 
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In  the  delayed  feedback  conditions,  subjects  performed  the  task  by  first  clicking  the  buttons 
sequentially  and  then  clicking  the  “when-done”  button  (see  Figure  6).  The  buttons  pushed  were 
displayed  on  the  screen  and  this  listing  was  erased  when  the  when-done  button  was  clicked. 
Subjects  could  also  restart  the  task  by  clicking  the  “start-over”  button.  Feedback  was  provided  only 
after  a  click  on  the  when-done  button.  The  subjects  could  not  proceed  to  the  next  task  until  the 
sequence  of  steps  had  been  entered  without  errors. 


Figure  6.  Sample  screen  of  practice  item  from  delayed  feedback  conditions. 


In  the  delayed  confirmatory  feedback  condition,  the  subjects  were  shown  two  lists  (see  Figure 
7)  of  the  correct  steps:  (1)  in  the  correct  order  and  (2)  in  the  incorrect  order.  The  example  in  Figure 
7  shows  that  the  subject  performed  Call  Forwarding  Program  steps  1 , 2, 6,  and  7  correctly  and  steps 
3, 4,  and  5  incorrectly.  The  last  column  on  the  right  of  the  display  presents  subject  entries  that  are 
correct  responses  entered  in  the  wrong  order  such  as  Touch  *12  and  Listen  For  Tone.  The  feedback 
was  erased  and  the  subject  was  asked  to  try  the  task  again. 

In  the  delayed  corrective  feedback  condition,  feedback  for  errors  consisted  of  a  presentation  of 
the  entire  correct  sequence  of  steps  for  that  task.  When  a  subject  made  an  error,  the  feedback 
consisted  of  a  hand  pointing  to  the  correct  sequence  of  steps.  Figure  8  shows  an  error  the  subject 
made  while  performing  the  Call  Forwarding  Program  task.  After  clicking  the  when-done  button, 
an  animated  hand  shows  the  subject  the  entire  sequence  of  correct  steps. 
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Presentation  Window 


Correct  Steos  Entered 

tn  the  Correct  Order 

Are  Listed  Below; 


CALL  FORWARDING 

Please  Forward  All 


STEP 

1  0-DPTT  ggEtSOWglS 

2  ILOEma  PTM2  TTEJCS1 

3  wrong 

4  wrong 

5  wrong 

6  OSHmill  IM3S  TTSEE 

7  H  m 


Return 


CLICK  HERE 
To  Continue 


Vour  Calls  To  Extension 
Number  37878 _ 

OTHER  CORRECT  STEPS 
(Bat  Wot  Entered  la  Correct  order) 

LISTEN  FOR  TONE 

TOUCH  37878 
TOUCH  *12 


Program  Codes 
Activate  Fowording  =  *12 


Figure  7.  Sample  screen  of  delayed  confirmatory  feedback. 


Presentation  Window 

INCORRECT  THE  CORRECT  CALL  FQRWARPIN5 

SEQUENCE  IS:  Please  Forward  All 

_  Vour  Colls  To  Extension 

Lxit 

Zb 

Listen  W 
—  Toi** 

n  n  nrwrm  mwnuwnruwm  Number  37878 

tULOinr  {SEBEDVIS2 

ajLiiaraa  poua  WDoa 

SeitcJvSooi 

_ 

f  ■—  \ 

Hong  Up 

1 

*§= 

Program  Codes 

GHI 

4 

f 

mno 

6 

P^S 

T 

* 

OPER 

0 

& 

Activote  Fowarding  =  *12 

Figure  8.  Sample  screen  of  delayed  corrective  feedback. 


Analysis 


Five  subjects  were  excluded  from  the  analysis  of  data;  three,  because  of  computer  failures;  and 
two,  because  they  did  not  complete  the  CBT.  Four  of  these  subjects  were  from  the  delayed 
confirmatory  conditions,  and  the  other  was  from  the  delayed  corrective  condition. 

Pre-questionnaire 

The  variables  analyzed  from  the  pre-questionnaire  included  age,  rank,  months  in  the  Navy, 
months  in  the  present  assignment,  microcomputer  experience,  video  game  experience,  mood,  the 
number  calls  made  or  answered  per  day,  and  the  number  of  CATS  features  the  subject  knew  how 
to  use.  Rank  ranged  from  one  to  nine  where  one  equals  the  military  rank  of  E- 1  and  nine  equals  the 
military  rank  of  E-9.  Microcomputer  and  video  game  experience  was  rated  on  a  5-point  scale  where 
1  =  none  and  5  =  much.  Mood  was  rated  on  a  5-point  scale  where  1  =  tired  and  5  =  alert.  These 
variables  were  analyzed  using  a  one-way  analysis  of  variance  (ANOVA)  comparing  all  five 
groups.  A  priori  contrasts  were  performed  for  the  main  effects  (i.e.,  immediate  vs.  delayed, 
confirmatory  vs.  corrective,  treatments  vs.  control).  Post-comparisons  among  the  five  group  means 
were  performed  using  t  tests. 

Computer-based  Training  Performance 

During  the  CBT  practice,  the  computer  recorded  the  number  of  errors  and  the  time  used  for 
each  task.  In  the  delayed  feedback  conditions,  the  computer  also  recorded  the  number  of  clicks  on 
the  when-done  and  start-over  buttons.  The  time  to  perform  all  the  tasks  was  totaled  and  converted 
to  seconds  for  each  subject. 

During  the  performance  test,  the  computer  recorded  every  response  (i.e.,  any  clicks  on  the 
graphic  simulation  of  the  phone)  and  created  a  protocol  for  each  subject.  The  protocol  was  a 
chronological  listing  of  which  button  was  clicked,  when  it  was  clicked,  and  the  number  of  the  test 
item.  A  response  to  a  test  item  was  scored  as  correct  if  it  matched  the  sequence  to  perform  the  task 
specified  in  that  test  item.  The  performance  test  score  was  the  number  of  correct  responses.  The 
time  to  complete  all  items  was  totaled  and  converted  to  seconds. 

An  additional  measure  of  performance,  a  modified  performance  test  score,  was  generated  by 
giving  credit  for  nearly  correct  responses.  For  all  test  items,  a  response  was  re-scored  as  correct  if 
the  error  involved  listening  for  a  tone.  In  addition,  responses  to  test  items  for  Program  Call 
Forwarding,  Use  Call  Forwarding,  and  Program  Abbreviate  Dialing  were  re-scored  as  correct  if  the 
subject  forgot  to  hang  up  the  phone.  These  particular  errors  were  re-scored  as  correct  because,  if 
they  occurred  during  the  operation  of  the  actual  CATS  equipment,  these  procedures  might  still 
work. 

The  CBT  performance  variables  were  analyzed  using  a  one-way  ANOVA.  The  contrasts  and 
the  range  tests  were  the  same  as  those  used  with  the  pre-questionnaire  variables.  Performance  test 
means  were  compared  for  all  groups.  CBT  practice  means  were  compared  for  only  the  treatment 
groups. 
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Post-questionnaire 


Answers  to  the  open-ended  questions  about  the  purpose  of  CATS  elements  (e.g.,  dial  tone, 
switch-hook)  were  scored  on  a  3-point  scale.  Incorrect  or  blank  answers  received  1  point;  answers 
that  were  specific  to  just  one  of  the  CATS  features  received  2-points;  answers  that  described  a 
purpose  that  could  apply  to  all  the  CATS  features  received  3-points.  Answers  to  the  open-ended 
comments  question  were  scored  as  either  positive  or  negative. 

The  quantitative  variables  analyzed  from  the  post-questionnaire  included  the  number  of  CATS 
features  the  subjects  thought  they  knew,  and  the  number  of  features  which  caused  frustration.  Other 
variables  included  self-ratings  of  mood  (on  a  5-point  scale)  where  1  =  tired  and  5  =  alert,  CBT  ease 
of  use  where  1  =  easy  and  5  =  difficult,  and  CBT  satisfaction  where  1  =  miserable  and  5  = 
enjoyable.  Responses  to  general  questions  about  CATS  procedures,  CBT  feedback,  and  using  CBT 
to  teach  control  panel  operation  were  scored  as  yes  or  no.  Again,  group  means  of  these  variables 
were  compared  using  a  one-way  ANOVA. 


RESULTS 


Total  Sample 

Pre-questionnaire 

Table  2  summarizes  the  biodemographic  and  self-rating  variables  from  the  pre-questionnaire 
for  the  total  sample  by  experimental  group.  Means  are  presented  for  age,  rank,  months  in  the  Navy, 
months  at  the  present  assignment,  microcomputer  experience,  video  game  experience,  and  mood. 
The  mean  number  of  times  a  day  that  subjects  answered  or  made  a  call  and  the  mean  number  of 
CATS  features  that  subjects  knew  how  to  use  are  also  shown.  No  statistically  significant 
differences  among  the  experimental  groups  were  found. 

Overall,  these  subjects  can  be  described  as  young  and  inexperienced  with  electronic  devices. 
Most  of  the  subjects  were  under  22  years  of  age  and  had  been  in  the  Navy  less  than  a  year.  Although 
most  of  the  subjects  were  familiar  with  video  games,  they  had  limited  experience  with 
microcomputers.  These  subjects  reported  little  phone  use  and  most  of  them  reported  knowing  how 
to  activate  only  one  CATS  feature. 
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Table  2 


Total  Sample:  Means  of  Biodemographic  and  Self-rating  Variables 


Feedback  Group 


Confirmatory 

Correct  Response 

Variable 

Immediate 

Delayed 

Immediate 

Delayed 

Control 

Age  (years) 

21.2 

20.7 

22.4 

20.0 

20.7 

Rank 

2.6 

2.5 

3.0 

2.8 

2.5 

Months  in  the  Navy 

8.6 

10.0 

12.3 

8.6 

9.8 

Months  at  Present 
Assignment 

4.4 

2.6 

3.7 

3.4 

3.0 

Microcomputer 

Experience 

2.2 

2.1 

2.1 

2.1 

2.1 

Video  Game 
Experience 

3.9 

4.2 

3.4 

3.9 

3.4 

Mood 

3.8 

4.0 

3.5 

4.0 

3.4 

Number  of  Times  a 
Day  Makes  a  Call 

0.9 

0.8 

1.1 

1.6 

1.3 

Number  of  Times  a 
Day  Answers  a  Call 

2.1 

1.8 

1.9 

4.8 

4.3 

Number  of  CATS 
Features  Subject 
Knows 

0.9 

0.6 

0.6 

0.9 

1.0 

n 

16 

13 

16 

14 

16 

Note.  Scores  for  those  who  completed  the  performance  test  (N  =  75).  Rank  ranges  from  one  to  nine  where  1  =  E-l  and 
9  =  E-9.  Computer  and  video  game  experience  are  on  a  5-point  scale  where  1  =  none  and  5  =  much.  Mood  is  on  a  5- 
point  scale  where  1  =  tired  and  5  =  alert.  Number  of  possible  CATS  features  is  eight 

Computer-based  Training  Performance 

Table  3  presents  the  means  and  standard  deviations  for  performance  test  score  by  experimental 
group  and  shows  a  main  effect  for  timing  of  feedback:  The  delayed  feedback  groups  significantly 
outperformed  the  immediate  feedback  groups  on  the  performance  test  score  (r(70)  =  4.58,  p  <  .01) 
and  on  the  modified  performance  test  score  (f(70)  =  3.65,  p  <  .01 ).  Likewise,  the  delayed  feedback 
groups  significantly  outperformed  the  control  group  on  the  performance  test  score  (r(70)  =  5.76,  p 
<  dl)  and  on  the  modified  performance  test  score  (r( 70)  =  7.94, p  <  .01).  The  immediate  feedback 
groups  significantly  outperformed  the  control  group  only  for  the  modified  performance  test  score 
(f(70)  =  5.07,  p  <  .01).  No  effects  were  found  for  content  of  feedback  (i.e.,  confirmatory  vs. 
corrective). 
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Table  3 


Total  Sample:  Means  and  Standard  Deviations  of  Performance  Test 


Feedback  Group 

Confirmatory 

Corrective 

Variable 

Immediate 

Delayed 

Immediate 

Delayed 

Control 

Performance  Test 

Score 

1.4 

3.1 

1.4 

2.9 

0.6 

(1.3) 

(1-4) 

(1.3) 

(1.9) 

(0.8) 

Modified  Performance 

Test  Score3 

4.4 

6.1 

4.1 

5.9 

1.4 

(1.9) 

(1.6) 

(2.2) 

(2.0) 

(1.3) 

Time  to  Complete 

Performance  Test 

729 

573 

712 

529 

680 

(seconds) 

(172) 

(127) 

(223) 

(129) 

(226) 

n  16 

13 

16 

14 

16 

Note.  Maximum  score  =  8.  Standard  deviations  are  given  in  parentheses. 

*Errors  involving  Listen  for  Tone  and  hanging  up  the  receiver  at  end  of  task  are  ignored. 

Table  3  also  presents  the  means  and  standard  deviations  for  time  to  complete  the  performance 
test  by  experimental  group.  Again,  a  main  effect  was  found  for  timing  of  feedback  but  not  for 
content:  The  delayed  feedback  groups  took  significantly  less  time  to  complete  the  performance  test 
than  the  immediate  feedback  groups  (t(70)  =  3.52,  p  <  .01).  Moreover,  the  delayed  feedback  groups 
used  significantly  less  time  than  the  control  group  (t(70)  =  2.22,  p  <  .05). 

Table  4  presents  the  means  and  standard  deviations  for  the  time  to  complete  practice.  In  the 
confirmatory  conditions,  the  immediate  feedback  group  used  19.4  minutes  compared  to  42.1 
minutes  used  by  the  delayed  group.  In  the  corrective  conditions,  the  immediate  feedback  group 
used  16.4  minutes  compared  to  28.4  minutes  used  by  the  delayed  group.  The  difference  between 
the  immediate  and  the  delayed  feedback  groups  was  significant  (t(55)  =  9.70,  p  <  .01 ).  In  addition, 
the  confirmatory  delayed  feedback  group  used  significantly  more  time  than  the  corrective  delayed 
feedback  group  (r(55)  =  5.21,  p  <  .01).  Table  4  also  presents  the  means  and  standard  deviations  for 
the  number  of  errors  committed  during  practice.  Table  4  also  indicates  that  the  delayed  groups 
committed  significantly  more  errors  than  the  immediate  groups  (f(55)  =  9.36,  p  <  .01). 
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Table  4 


Total  Sample:  Means  and  Standard  Deviations  of  Practice  Variables 


Variable 

Feedback  Group 

Confirmatory 

Correct 

Immediate 

Delayed 

Immediate 

Delayed 

Practice  Time 

1164 

2528 

982 

1703 

(seconds) 

(231) 

(639) 

(263) 

(443) 

Practice  Errors 

44.9 

151.5 

25.8 

122.2 

(22.3) 

(66.9) 

(11.7) 

(49.2) 

n 

16 

13 

16 

14 

Note.  Standard  deviations  are  given  in  parentheses. 

Post-questionnaire 

The  post-questionnaire  consisted  of  open-ended  questions  about  the  purpose  of  CATS 
elements  and  self-ratings  of  mood,  CBT  ease  of  use,  and  CBT  satisfaction.  The  open-ended 
questions  were  designed  to  elicit  the  subjects’  semantic  understanding  of  the  CATS  buttons  and 
switches.  Semantic  knowledge  is  knowing  the  purpose  of  pressing  an  individual  switch  or  button. 
The  subjects  were  asked  to  describe  the  purpose  of  the  switch-hook,  dial  tone,  the  *  button,  and  the 
#  button.  The  scores  for  these  questions  were  totaled;  the  means  and  standard  deviations  are  shown 
in  Table  5.  Although  the  immediate  corrective  feedback  group  appears  to  have  the  lowest 
understanding,  no  significant  differences  were  found  for  understanding  the  purpose  of  CATS 
elements. 

Generally,  no  significant  differences  were  found  for  the  self-rating  items,  for  mood,  and  for  the 
number  of  CATS  features  that  caused  subjects  to  feel  mastery  or  frustration.  However,  significant 
differences  were  found  for  CBT  satisfaction.  Table  5  shows  the  means  and  standard  deviations  for 
CBT  satisfaction,  and  indicates  that  the  subjects  in  the  confirmatory  feedback  groups  enjoyed  the 
training  significantly  more  than  the  subjects  in  the  corrective  feedback  groups  (F(l,53)  =  8.00, p  < 
.01).  Yet,  no  significant  differences  were  found  among  the  feedback  groups  for  how  easy  the  CBT 
was  to  use  (see  Table  5). 

The  post-questionnaire  also  contained  a  few  open-ended  questions  about  employing  CBT  to 
teach  how  to  operate  a  PEP  device.  The  answers  to  these  questions  were  generally  positive. 
Approximately  93  percent  of  the  subjects  were  happy  with  the  feedback,  93  percent  recommended 
that  CBT  be  used  to  teach  control  panel  operation,  and  73  percent  wrote  positive  comments  (while 
only  5  percent  wrote  negative  comments)  to  the  open-ended  asking  for  comments  question.  Again, 
no  significant  differences  were  found  among  the  feedback  groups  for  these  measures. 
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Table  5 


Total  Sample:  Means  and  Standard  Deviations  of  Post-questionnaire  Items 


Feedback  Group 

Confirmatory 

Corrective 

Item 

Immediate 

Delayed 

Immediate 

Delayed 

Understanding  CATS 
Elements 

10.12 

9.92 

9.00 

10.07 

(1.33) 

(1.71) 

(2.22) 

(1-77) 

CBT  Satisfaction 

4.19 

4.17 

3.75 

3.46 

(0.65) 

(0.72) 

(0.77) 

(0.88) 

CBT  Ease  of  Use 

2.00 

2.46 

2.56 

2.57 

(1.15) 

(1.45) 

(1.09) 

(0.94) 

n 

16 

13 

16 

14 

Note.  For  understanding  CATS  elements,  minimum  rating  score  =  4  and  maximum  rating  score  =12.  For  CBT 
satisfaction  and  CBT  ease  of  use,  minimum  rating  score  =  1  and  maximum  rating  score  =  5.  Standard  deviations  are 
given  in  parentheses. 

Moderator  Variables 

Ability  Level 

The  subjects  were  drawn  from  two  populations  of  Navy  students,  those  attending  ASW  training 
at  the  Fleet  ASW  Training  Center  Pacific  and  those  attending  apprenticeship  training  at  the  RTC. 
Since  the  ASW  students  generally  have  higher  achievement  scores  than  the  RTC  students,  it  was 
expected  that  the  ASW  students  would  perform  better  on  the  CBT  tasks  than  the  RTC  students.  A 
2X2X2  ANOVA  (sample  by  timing  by  content)  was  performed  on  the  performance  measures. 
Table  6  presents  the  means  and  standard  deviations  for  performance  test  score,  and  time  to 
complete  performance  test,  and  practice  time  by  sample.  The  ASW  sample  significantly 
outperformed  the  RTC  sample  on  the  performance  test  (F(l,51)  =  16.18,  p  <  .01),  and  used 
significantly  less  time  to  complete  it  (F(l,51)  =  7.64,  p  <  .01)  and  the  practice  (F(l,51)  =  8.39,  p  < 
.01).  However,  in  spite  of  this  strong  main  effect  for  ability  level  (i.e.,  ASW  sample  vs.  RTC 
sample),  there  were  no  significant  interactions  for  sample  by  timing,  sample  by  content,  and 
sample  by  timing  by  content.  Thus,  ability  level  appears  to  have  little  impact  on  the  relationship 
between  type  of  feedback  and  CBT  performance. 
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Table  6 


Means  and  Standard  Deviations  of  Performance  by  Sample 


Sample 

Performance 

Test  Score 

Performance 

Test  Time 

Practice  Time 

n 

M 

SD 

M 

SD 

M 

SD 

Immediate  Feedback  Conditions 

ASW  Students 

1.7 

1.3 

655 

161 

972 

206 

22 

RTC  Students 

0.7 

0.8 

864 

196 

1293 

235 

10 

Delayed  Feedback  Conditions 

ASW  Students 

3.6 

1.5 

539 

95 

1996 

665 

19 

RTC  Students 

1.6 

1.3 

576 

190 

2347 

696 

8 

Note.  Maximum  test  score  =  8.  Time  is  in  seconds. 

The  finding  that  ability  level  does  not  affect  the  role  of  feedback  can  be  illustrated  by 
examining  the  significant  differences  among  the  treatment  groups  for  the  ASW  sample.  The  means 
and  standard  deviations  for  the  performance  test  score  for  the  ASW  sample  (Table  7)  and  for  the 
total  sample  (Table  3)  show  a  similar  main  effect  for  timing  of  feedback.  The  delayed  feedback 
groups  for  the  ASW  sample  significantly  outperformed  the  immediate  feedback  groups  on  the 
performance  test  score  (/(46)  =  4.78,  p  <  .01),  and  on  the  modified  performance  test  score  (r(46)  = 
3.95 ,p  <  .01).  Likewise,  the  delayed  feedback  groups  significantly  outperformed  the  control  group 
on  the  performance  test  score  (f(46)  =  6.21,  p  <  .01),  and  on  the  modified  performance  test  score 
(r(46)  =  9.15,  p  <  .01).  Again,  no  effects  were  found  for  content  of  feedback. 

Table  7 

Means  and  Standard  Deviations  of  Performance  Test  Scores:  ASW  Sample 


Variable 

Feedback  Group 

Confirmatory 

Corrective 

Control 

Immediate 

Delayed 

Immediate 

Delayed 

Performance  Test 

Score 

1.7 

3.8 

1.7 

3.5 

0.5 

(1.3) 

(1.2) 

(1.3) 

(1.7) 

(0.7) 

Modified  Performance 

test  Score3 

4.7 

6.9 

4.7 

6.4 

1.1 

(1.8) 

(0.9) 

(1.9) 

(1.4) 

(1.3) 

n 

11 

9 

11 

10 

10 

Note.  Maximum  score  =  8.  Standard  deviations  are  given  in  parentheses. 

'Errors  involving  Listen  for  Tone  and  hanging  up  the  receiver  at  end  of  task  are  ignored. 
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Practice  Time 

Since  the  subjects  in  the  delayed  feedback  conditions  took  longer  to  complete  the  practice  than 
the  subjects  in  the  immediate  feedback  conditions,  it  is  possible  that  the  strong  performance  effects 
found  for  delayed  feedback  could  be  moderated  by  practice  time.  A  2  X  2  (timing  by  content) 
analysis  of  covariance  (ANCOVA)  (with  practice  time  as  the  covariate)  was  performed  on  the 
modified  performance  test  score.  After  accounting  for  the  covariate,  significant  main  effects  were 
still  found  for  the  timing  of  feedback  (F(l,54)  =  1 1.24,  p  <  .0)1,  and  no  significant  effects  were 
found  for  content.  The  correlation  between  practice  time  and  performance  was  not  significant,  (r  = 
.216,  r(57)  =  1.67).  The  mean  modified  performance  test  scores  and  the  mean  modified 
performancetest  scores  adjusted  for  practice  time  are  summarized  by  timing  of  feedback  in  Figure 
9.  The  pattern  of  results  for  the  means  adjusted  by  the  covariate  is  very  similar  to  that  obtained  with 
the  mean  modified  performance  test  score.  The  difference  between  the  performance  of  the  subjects 
who  received  immediate  and  delayed  feedback  seems  to  be  slightly  larger  when  the  means  were 
adjusted  by  the  covariate.  This  would  suggest  that  practice  time  and  performance  are  negatively 
correlated  within  each  feedback  timing  group,  and,  in  fact,  the  correlation  between  practice  time 
and  performance  is  -.207  for  the  immediate  feedback  subjects  and  -.132  for  the  delayed  feedback 
subjects. 


Note.  Errors  involving  Listen  for  Tone  and  hanging  up  the  receiver  at  end  of  task  are  ignored. 
Figure  9.  Mean  modified  performance  test  score  by  feedback  treatment 


22 


Task  Difficulty 

The  eight  to-be-leamed  tasks  presented  by  the  CBT  varied  in  difficulty.  The  length  of  the  tasks 
ranged  from  3  to  1 1  steps  (see  Table  1).  The  performance  test,  which  comprised  these  eight  tasks, 
was  divided  into  two  subscales:  long  items  and  short  items.  Long  items  had  seven  or  more  steps 
while  short  items  had  fewer  than  seven  steps.  Each  of  these  subscales  contained  four  items.  A  2  X 
2X2  ANOVA  was  performed.  The  between-subject  factors  were  feedback  timing  and  feedback 
content,  and  the  within-subject  factor  was  item  length  (long  vs.  short).  Figure  10  presents  the 
modified  performance  test  mean  scores  for  ail  conditions  and  indicates  that  the  delayed  feedback 
groups  significantly  outperformed  the  immediate  feedback  groups  on  both  the  long  and  short  items 
(F(l,55)  =  22.36, p  <  .01).  However,  a  significant  interaction  was  found  between  feedback  timing 
and  item  length  (F(l,55)  =  4.1 1  ,p  <  .05).  The  advantage  found  for  the  delayed  feedback  groups  is 
much  stronger  for  the  long  items  than  the  short  items. 


Note.  Errors  involving  Listen  for  Tone  and  hanging  up  the  receiver  at  end  of  task  are  ignored. 
Figure  10.  Mean  modified  performance  test  score:  Long  vs.  short  items. 
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DISCUSSION 


General 

It  was  hypothesized  that,  when  using  CBT  to  learn  a  PEP  task,  subjects  provided  with 
confirmatory  feedback  will  outperform  those  provided  with  corrective  feedback,  and  subjects 
provided  with  delayed  feedback  will  outperform  those  provided  immediate  feedback.  On  the 
surface,  these  results  provide  little  support  for  the  first  hypothesis.  No  performance  effects  were 
found  when  the  content  of  CBT  feedback  was  manipulated.  However,  these  results  indirectly 
support  the  use  of  confirmatory  feedback  over  corrective  feedback.  Both  confirmatory  feedback 
and  corrective  feedback  were  equally  effective  because  both  showed  students  the  results  of  their 
actions.  It  appears  that  providing  an  indication  that  a  response  is  wrong  is  central  to  learning 
procedures.  This  research  suggests  that  just  adding  the  correct-response  to  the  student’s  knowledge 
of  results  does  not  improve  learning  device  sequences.  In  other  words,  these  results  provide  little 
support  for  including  the  correct-response  as  part  of  the  feedback  provided  for  student  errors. 

Strong  support  was  found  for  the  second  hypothesis.  Delayed  feedback  increased  learning  as 
measured  by  the  performance  test  scores.  Moreover,  the  delayed  feedback  groups  completed  the 
performance  test  significantly  faster.  The  delayed  feedback  was  particularly  effective  for  subjects 
learning  the  conceptually  complex  features.  Delayed  feedback  may  have  created  an  instructionless 
learning  environment  which  increased  learning.  The  subjects  in  the  delayed  feedback  conditions 
performed  better  because  they  had  acquired  a  richer  device  schema  of  how  to  operate  the  PEP 
device. 

Shrager  and  Klahr  (1986)  found  that  during  instructionless  learning,  subjects  formed  a 
conceptual  model  or  a  device  schema  of  a  PEP  device.  Their  learners  acquired  a  schema  for 
operating  a  programmable  toy  tank  which  consisted  of  knowledge  about  “the  syntax  of  interaction, 
the  semantics  of  various  functions,  and  the  contents  of  the  device  which  mediate  interaction”  (p. 
176).  A  highly  developed  device  schema  is  one  which  contains  both  syntactic  knowledge  and 
semantic  knowledge.  However,  none  of  the  results  from  this  research  provide  any  evidence  that  the 
delayed  feedback  subjects  acquired  a  more  highly  developed  device  schema. 

Consequently,  the  data  were  reanalyzed  to  identify  the  sequences  of  steps  indicative  of  either 
syntactic  knowledge  or  semantic  knowledge.  Sequences  of  steps  which  implied  syntactic 
knowledge  were  defined  as  those  that  were  highly  associated  in  time  and  space.  The  CATS  tasks 
were  analyzed  step  by  step  and  the  probabilities  of  how  often  one  step  followed  another  were 
calculated.  The  two  most  highly  associated  sequences  in  the  eight  CATS  tasks  were  Lift  Receiver 
followed  by  Listen  For  Tone  and  Press  Switch-hook  followed  by  Listen  For  Tone.  Listen  For  Tone 
always  follows  Lift  Receiver,  and  Listen  For  Tone  follows  Press  Switch-hook  in  five  out  of  six 
occurrences.  These  steps  have  a  high  probability  of  occurring  one  after  the  other. 

Steps  which  represent  a  single  function  imply  the  acquisition  of  semantic  knowledge.  For 
example,  the  concept  of  programming  is  implied  when  the  #  button  follows  the  extension  number 
in  the  Abbreviated  Dialing  Program  task.  It  was  predicted  that  subjects  in  the  immediate  feedback 
groups  would  acquire  only  syntactic  knowledge  while  subjects  in  the  delayed  feedback  groups 
would  acquire  both  syntactic  and  semantic  knowledge. 
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The  performance  test  protocols  for  each  feedback  group  were  reexamined  for  highly  associated 
steps.  The  targeted  sequences  were  the  Lift  Receiver  followed  by  Listen  For  Tone  in  the  Call 
Pickup  and  Abbreviated  Dialing  Use  tasks,  and  the  Press  Switch-hook  followed  by  Listening  for 
Tone  in  the  Call  Transfer  task.  Table  8  presents  the  response  frequencies  for  the  targeted 
sequences.  Table  8  indicates  that  the  subjects  in  both  the  immediate  and  delayed  feedback  groups 
almost  always  remembered  that  Listen  For  Tone  follows  Lift  Receiver,  and  that  20  of  the 
immediate  feedback  subjects  (63%)  and  19  of  the  delayed  feedback  subjects  (70%)  remembered 
that  Listen  for  Tone  follows  Press  Switch-hook  during  the  Call  Transfer  task.  This  difference  was 
not  significant. 

It  can  be  argued  that  the  sequences  in  Table  8  are  also  semantically  related.  That  a  tone  follows 
either  Lift  Receiver  or  Press  Switch-hook  may  imply  the  concept  of  securing  an  open  phone  line. 
Therefore,  remembering  that  a  tone  follows  lifting  the  receiver  or  pressing  the  switch-hook  may 
indicate  either  syntactic  or  semantic  knowledge  or  both. 

Table  8 

Response  Frequencies  for  Highly  Associated  Steps 


Response  Frequency 

Steps/Feedback  Group 

Correct 

Incorrect11 

n 

Lift  Receiver  before  Listen  For  Tone 
(In  Call  Pickup) 

Immediate 

32 

0 

32 

Delayed 

27 

0 

27 

Lift  Receiver  before  Listen  For  Tone 
(In  Abbreviated  Dialing  Use) 

Immediate 

31 

1 

32 

Delayed 

26 

1 

27 

Press  Switch-hook  before  Listen  for  Tone 
(In  Call  Transfer) 

Immediate 

20 

12 

32 

Delayed 

19 

8 

27 

'Includes  responses  where  first  step  is  omitted. 

The  performance  test  protocols  were  then  reexamined  for  semantically  related  steps  that  are  not 
highly  associated.  Two  sequences  were  targeted:  Enter  program  code  *17  (i.e.,  activate  abbreviated 
list)  followed  by  enter  19  (i.e,  the  list  number)  in  the  Abbreviated  Dialing  Program  and 
Abbreviated  Dialing  Use  tasks,  and  enter  35656  (i.e.,  an  extension  number)  followed  by  the 
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#  button  in  the  Abbreviated  Dialing  Program  task.  The  most  probable  step  following  a  program 
code  is  Listen  For  Tone  which  occurs  50  percent  of  the  time  for  all  tasks.  The  probability  of  a 
number  following  a  program  code  is  only  25  percent.  The  most  probable  steps  following  an 
extension  number  are  either  Press  Switch-hook  or  Listen  For  Tone,  which  occurs  about  67  percent 
of  the  time.  The  probability  of  the  #  button  following  an  extension  number  is  only  about  16  percent. 

Table  9  presents  the  frequencies  of  correct  responses,  high  probability  errors,  and  low 
probability  errors  for  these  semantically  related  steps.  Low  probability  errors  are  any  errors  not 
described  (above)  as  the  most  probable.  Only  12.5  percent  of  the  immediate  feedback  subjects 
remembered  that  step  19  follows  step  *17  during  the  Abbreviated  Dialing  Program  task.  Most  of 
the  subjects  in  this  group  followed  the  program  code  with  a  high  probability  error  steps  (i.e..  Listen 
For  Tone).  In  contrast,  over  59  percent  of  the  delayed  feedback  subjects  remembered  the  sequence 
correctly,  which  is  significantly  higher  than  the  frequency  reported  for  the  immediate  feedback 
group  (%*(2)  =  14.90,  p  <  .01). 


Table  9 

Response  Frequencies  for  Semantically  Related  Steps 


Response  Frequency 

Steps/Feedback  Group 

High  Probability 

Low  Probability 

Correct 

Error 

Error3 

n 

Step  *17  before  Step  19 

(In  Abbreviated  Dialing  Program) 

Immediate 

4 

17 

11 

32 

Delayed 

16 

5 

6 

27 

Step  *17  before  Step  19 

(In  Abbreviated  Dialing  Use) 

Immediate 

7 

16 

9 

32 

Delayed 

19 

5 

3 

27 

Step  35656  before  Step  # 

(In  Abbreviated  Dialing  Program) 

Immediate 

12 

13 

7 

32 

Delayed 

22 

2 

3 

27 

'Includes  responses  where  first  step  is  omitted. 

During  the  Abbreviated  Dialing  Use  task,  22  percent  of  the  immediate  feedback  subjects 
correctly  remembered  that  a  list  number  follows  the  activate  Abbreviated  Dialing  List  Program 
code.  Most  of  the  subjects  in  the  immediate  feedback  conditions  followed  the  program  code  with 
a  high  probability  error.  In  contrast,  70  percent  of  the  subjects  in  the  delayed  feedback  conditions 
remembered  this  sequence  correctly.  This  difference  was  significant  QC2(2)  =  13.98,  p  <  .01). 
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Similar  results  were  found  for  the  other  semantical  sequence  occurring  during  the  Abbreviated 
Dialing  Program  task.  Significantly  more  subjects  in  the  delayed  feedback  conditions  remembered 
that  the  #  button  follows  the  extension  number  than  did  those  in  the  immediate  feedback  conditions 
(X2(2)  =  12.27,/?  <  .01).  Again,  the  most  likely  response  for  the  immediate  feedback  subjects  was 
to  commit  a  high  probability  error.  Thus,  subjects  in  the  immediate  feedback  conditions  tended  to 
provide  the  syntactically  probable  response  even  when  it  might  be  incorrect  while  subjects  in  the 
delayed  feedback  conditions  provided  the  syntactically  probable  response  only  if  the  steps  were 
semantically  consistent.  Moreover,  subjects  in  the  delayed  feedback  conditions  provided 
semantically  accurate  responses  even  when  it  was  not  probable  syntactically. 

In  summary,  subjects  in  both  feedback  timing  groups  learned  steps  that  were  related 
syntactically,  but  only  the  subjects  in  the  delayed  feedback  conditions  remembered  steps  that  were 
related  semantically.  These  results  suggest  that  subjects  in  the  delayed  feedback  conditions 
acquired  both  syntactic  and  semantic  knowledge  about  the  PEP  device  while  subjects  in  the 
immediate  feedback  conditions  acquired  only  syntactic  knowledge.  This  analysis  provides 
additional  evidence  that  delaying  the  feedback  for  student  errors  during  CBT  practice  was 
beneficial  in  the  development  of  a  usable  device  schema. 

Implications 

The  results  of  this  study  seem  to  run  counter  to  the  role  of  feedback  in  computerized  instruction 
promoted  by  many  instructional  design  textbooks.  Instructional  designers  generally  believe  that 
feedback  in  computerized  instruction  should  be  immediate.  In  their  Handbook  of  Computer-Based 
Training,  Dean  and  Whitlock  (1983)  state  that  “once  the  learner  has  made  his  response,  he  should 
receive  feedback  on  the  adequacy  of  his  performance”  (p.  58).  This  belief  is  based  on  research  in 
animal  behavior  (Skinner,  1968;  Thorndike,  1911)  and  in  programmed  instruction  (Bullock,  1978; 
Hartley,  1972;  Taber,  Glasser,  &  Schaefer,  1965).  Proponents  of  this  research  argue  that  immediate 
feedback  serves  to  modify  or  maintain  the  learner’s  actions  and  that  delayed  feedback  may  result 
in  little  or  no  learning. 

However,  in  this  study  of  CBT  for  complex  tasks,  immediate  feedback  was  not  found  to  be 
better  than  delayed  feedback.  In  fact,  as  the  conceptual  complexity  of  the  CBT  task  increased, 
immediate  feedback  groups  performed  considerably  worse  than  the  delayed  feedback  groups. 
Immediate  feedback  did  not  seem  to  encourage  the  development  of  a  usable  device  schema  for 
complex  tasks.  Immediate  feedback  may  be  adequate  for  learning  tasks  with  short  to-be-leamed 
sequences  that  activate  only  one  function.  Yet,  this  form  of  instruction  “does  not  result  in 
meaningful  learning,  but  only  rote  recall  of  information  presented”  (Jonassen,  1988;  p.  151). 
Training  approaches  such  as  drill  and  practice  that  feature  rote  recall  of  information  have  been 
found  not  to  be  an  effective  form  of  instructional  delivery  (Kulik,  Schwalb,  &  Kulik,  1982)  and 
have  been  reported  to  be  “dry,  boring,  and  unpleasant”  (Salisbury,  1988;  p.  103). 

Moreover,  instructional  designers  also  believe  that  content  of  feedback  in  CBT  should  be  the 
correct  response:  “When  an  answer  is  correct,  feedback  should  not  only  inform  learners  that  they 
are  wrong  but  should  also  provide  corrective  information”  (Steinberg,  1984;  p.  86).  They  maintain 
that  computerized  instruction  “should  do  more  than  simply  say  your  answer  is  incorrect”  (Willis, 
1987,  p.  163).  However,  this  study  found  no  particular  advantage  in  using  correct  response 
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feedback  in  learning  how  to  operate  a  complex  device  and  no  evidence  to  support  the  widely 
accepted  belief  that  CBT  should  provide  corrective  feedback  for  student  errors. 


CONCLUSIONS 

The  CBT  developed  in  this  study  was  an  effective  form  of  instruction,  which  is  consistent  with 
the  results  found  in  meta-analyses  of  computerized  instruction  (Bangert-Drowns,  C.  C.  Kulik,  &  J. 
A.  Kulik,  1985;  C.  C.  Kulik,  J.  A.  Kulik,  &  Bangert-Drowns,  1985;  J.  A.  Kulik,  &  C.  C.  Kulik, 
1988;  Niemiec  &  Walberg,  1987;  Roblyer,  1990).  Each  CBT  group  outperformed  the  control 
group,  which  received  no  CBT.  The  experimental  computerized  training  was  enjoyable  and  easy 
to  use.  Moreover,  this  research  has  provided  information  on  the  relative  effectiveness  of  various 
feedback  designs.  Feedback  was  found  to  be  the  most  effective  when  it  was  delayed  until  the  end 
of  the  button-pushing  task.  Delaying  the  feedback  was  particularly  effective  when  the  to-be- 
learned  tasl'  consisted  of  more  than  four  steps  and  involved  more  than  one  function. 

FUTURE  EFFORTS 


Research 

Retention 

Since  the  strong  advantage  found  for  delayed  feedback  may  only  apply  for  short  term  retention 
(i.e.,  recalled  immediately  after  the  practice),  future  studies  should  investigate  the  long  term  effects 
of  delayed  feedback.  For  example,  a  performance  test  could  be  administered  24  hours  after  the 
CBT.  Since  delayed  feedback  encourages  the  development  of  device  schema,  the  effects  found  for 
delayed  feedback  should  endure  for  more  than  just  the  training  session. 

Practice  Time 

Although  the  advantage  found  for  delayed  feedback  was  unrelated  to  the  amount  of  time  used 
during  practice,  subjects  spent  significantly  more  time  in  practice  in  the  delayed  feedback 
conditions.  Indeed,  increased  practice  time  may  be  a  desired  characteristic  in  the  design  of  CBT. 
However,  an  experiment  in  which  the  subjects  in  all  the  conditions  have  equal  practice  time  may 
provide  a  more  appropriate  test  of  feedback  in  CBT.  Future  studies  should  increase  the  practice 
time  for  the  immediate  feedback  conditions.  The  practice  time  could  be  increased  by  increasing  the 
number  of  trials  for  the  immediate  feedback  conditions.  Conversely,  practice  time  for  the  delayed 
feedback  conditions  could  also  be  reduced.  In  particular,  subjects  could  be  shown  how  to  do  the 
task  before  they  practice  the  task.  Other  experimental  variations  could  include  not  requiring 
subjects  to  start  at  the  beginning  of  the  task  each  time  they  make  an  error  and  not  erasing  the 
subjects’  responses  from  the  screen  during  feedback. 

Applications 

Computers  have  become  an  important  tool  in  helping  us  plan,  communicate,  and  calculate. 
Computers  are  also  becoming  important  in  helping  us  learn  because  they  can  present  a  body  of 
knowledge  consistently  and  conveniently.  These  findings  should  be  useful  to  those  who  design 
computerized  instruction  because  it  describes  the  benefits  of  different  types  of  feedback  designs. 
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The  lesson  developer  must  consider  other  factors  besides  effectiveness  when  designing  CBT. 
The  CBT  must  also  be  affordable,  cost  less  than  the  actual  equipment,  use  less  training  time  than 
normal  classroom  instruction,  and  be  easy  to  use.  For  instance,  delayed  confirmatory  feedback 
during  CBT  may  not  be  desirable  because  it  would  be  the  most  expensive  of  the  feedback  strategies 
to  develop.  It  requires  additional  computer  programming  to  understand  partially  correct  sequences. 
Delaying  feedback  during  a  CBT  lesson  may  also  not  be  desirable  because  it  might  increase  the 
length  of  the  lesson  beyond  the  time  allowed  for  training.  In  addition,  corrective  feedback  must  be 
employed  carefully  because  it  may  bore  some  users. 

These  circumstances  must  be  weighed  in  conjunction  with  effects  on  performance  when 
deciding  what  type  of  feedback  to  implement.  For  a  simple  to-be-leamed  device,  using  feedback 
which  consists  of  the  correct  response  or  is  provided  immediately  will  not  significantly  reduce  the 
effectiveness  of  CBT.  If  CBT  is  being  developed  for  a  device  that  is  as  complex  as  CATS,  then 
delaying  the  feedback  is  recommended.  These  findings  along  with  cost  and  feasibility  data  should 
help  the  lesson  developer  design  feedback  which  is  sensitive  to  both  the  training  requirements  and 
the  instructional  setting. 
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APPENDIX  A 
PRE-QUESTIONNAIRE 


A-0 


PRE-QUESTIONNAIRE 


Last  Four  Digits  of  Soda]  Security  Number _  Date: _ _ T>me:_ _ 

Rating: _  Age: _  Years  in  Navy: _  Months  at  NTC  (or  present  assignment): _ 

Previous  Duty  Station:  Number  of  Months  at  Previous  Duty  Sh  m: 

1.  How  many  times  a  dav  do  vou  make  a  call  nn  a  Maw  Phone? 

2.  How  many  times  a  dav  do  vou  answer  a  Mavy  Phone? 

3.  Please  place  a  check  mark  next  to  those  features  of  the  Navy  phone  system  you  know  how  to  use: 

Abbreviated  Dialing _  CaJ  Foiwaitfng  Call  PkSa*) _  Call  Waiting 

Conference  Calling _  Transferring  a  Cat,  Last  Number  Racial.  Can  Hold _ 

The  following  questions  ate  on  a  five  point  scale:  (Please  Circle) 


Tired 

Alert 

4. 

How  do  you  feel  right  now? 

1  2 

3 

4 

5 

None 

Much 

5. 

How  much  experience  do  you  have  with  microcomputers? 

1  2 

3 

4 

5 

None 

Much 

6. 

How  much  experience  do  you  have  with  video  games? 

1  2 

3 

4 

5 

A- 1 


1 


**  PRIVAuy  act  STATEMENT  ^  11  " . . 

The  Privacy  Act  of  1974  (Public  Law  93-579)  implements  5  U.S.C.  552A  (Privacy  Act  of  1974)  and  DOD  Directive  5400.1 1 
(Personal  Privacy  and  Rights  of  Individuals  Regarding  Their  Records)  and  is  pubfished  In  Title  32  C.F.R.,  Part  701. 
subparts  P  and  G.  This  Act  requires  us  to  provide  you  with  certain  information  about  the  research  before  we  ask  you  to 
give  us  information  about  yourself.  Ybu  must  be  toid  the  following: 


1 .  SECNAV  Instruction  5211 .5C,  ‘Personnel  privacy  and  rights  of  Individuais  regarding  records  pertaining  to 
themselves,*  Is  available  for  you  review. 

2.  The  brief  title  of  this  research  project  is:  *The  Role  of  Feedback  in  Computer-Based  Training* 

3 .  The  primary  purpose  of  this  research  is  to  improve  computer-based  training  systems  in  teaching  control 
panel  operation. 

4.  The  following  personal  information  win  be  requested:  age,  rank,  time  at  current  duty  station,  time  in 
service,  and  the  last  four  numbers  of  your  social  security  number.  The  last  four  numbers  of  your  social 
security  number  is  needed  only  for  matching  questionnaire  data  to  experimental  data  and  It  will  not  be 
used  to  obtain  information  about  your  service  record. 

5 .  The  information  collected  during  this  study  will  be  used  for  research  purposes  only.  It  will  not  be  given  to 
anyone  outside  this  center,  and  it  will  not  be  used  for  anything  besides  scientific  study.  The  information 
you  give  will  not  help  or  hurt  you  personally  in  any  way  during  your  tour  with  the  Navy. 

6.  M>ur  participation  in  this  study  is  entirely  voluntary.  Nothing  win  happen  to  you  if  you  decide  against 
participating  in  this  study. 


A- 2 


CONSENT  TO  ACT  AS  A  SUBJECT  FOR  SCIENTIFIC  RESEARCH 


| _ _ _ voluntarily  agree  to  serve  as  a  subject  for  scientific  research 

Involving  the  following: 

Learning  how  to  operate  the  program  entry  panel  (PEP)  for  the  Navy's  Consolidated  Area  Telephone 
System  (CATS).  I  understand  I  will  perform  PEP  tasics  using  a  PEP  simulated  on  a  computer  monitor. 

(2)  The  procedure  outlined  in  in  paragraph  (1 )  has  been  explained  to  me. 

(3)  I  understand  that  the  procedure  outlined  in  paragraph  (1)  may  involve  the  following  risks  and  discomforts: 
There  are  no  known  risks  In  this  procedure. 

(4)  I  understand  that  this  research  win  not  benefit  me  directly.  My  participation  In  this  project  will  Increase 
scientific  understanding  of  how  people  learn  how  to  operate  control  panels.  This  research  is  intended  to 
be  of  benefit  to  others  and  to  the  Navy. 

(5)  I  understand  that  any  questions  I  may  have  during  the  experiment  will  be  answered,  and  that  I  may  end  my 
participation  in  this  study  at  any  time. 

(6)  l  have  read  the  attached  Privacy  Act  Statement,  and  I  understand  that  giving  information  about  myself  is 
entirely  voluntary  and  that  the  uses  of  that  information  will  be  only  for  scientific  purposes. 


Signature 


Date 


APPENDIX  B 

CBT  INTRODUCTION  SCREENS 


B-0 


Presentation  Window 


Navg 


PertuMl 


R  &  D 


Center 


Presentation  Window 


How  To  Operate  A 
Program  Entry  Panel  (PEP) 


a 


An  Independent  Research 
Project  Sponsored  By  The 
Navy  Personnel  Research 
And  Development  Center 


Pnncpal  Investigator  and  Author  Michael  B.  Cowen 


B-l 


Presentation  Window 


USING  THIS  LESSON  REQUIRES 
THAT  YOU  KNOW  HOW  TO: 


A.  Point  The  Mouse 

B.  Click  The  Mouse 


The  Mouse  Pictured  To  The  Right 
Should  Be  Sitting  On  The  Right 
Side  Of  The  Computer 


Presentation  Window 


TO  POINT  THE  MOUSE,  JUST 
MOVE  THE  MOUSE  ON  THE  PAD. 
YOU'LL  SEE  THAT  A  POINTER 
MOVES  ON  THE  SCREEN. 


TO  CLICK  THE  MOUSE,  POINT  TO 
AN  OBJECT  AND  PRESS  AND 
RELEASE  THE  MOUSE  BUTTON. 


TO  CONTINUE,  CLICK  ON  THIS  BOX  »|  | 


B-2 


Presentation  Window 


GOOD. ...YOU  HAVE  LEARNED  HOW  TO  POINT 
AND  CLICK  THE  MOUSE. 

THE  OBJECT  OF  THIS  LESSON  IS  TO  TEACH 
YOU  HOW  TO  USE  THE  NAVY'S  CONSOLIDATED 
AREA  TELEPHONE  SYSTEM  (CATS). 

SPECIFICALLY  YOU  WILL  LEARN  HOW  TO 
OPERATE  THE  KEYPAD  FOR  THE  AT&T 
MODEL  2500  TELEPHONE. 


TO  CONTINUE.  CLICK  ON  THIS  BOX - ►£jetjjrr^ 


Presentation  Window 


YOU  WILL  LEARN  HOW  TC  USE  SIX  DIFFERENT 
FEATURES  OF  THE  NAVY’S  PHONE  SYSTEM: 


1 .  Abbreviated  Dialing 

2.  Call  Forwarding 

3.  Call  Pickup 

4.  Conference  Calling 

5.  Hold 

6.  Call  Transfer 


TO  CONTINUE.  CLICK  ON  THIS  BOX - ►fBetumJ 


B-3 


Presentation  Window 


THE  LESSON  CONSISTS  OF  PRACTICE  ITEMS  WHERE 
YOU  ARE  REQUIRED  TO  ENTER  KEYPRESSES  ON  A 
SIMULATION  OF  A  TELEPHONE  KEYPAD  USING  A 
MOUSE. 

IN  A  MOMENT  YOU  WILL  SEE  THE  SIMULATED  KEYPAD. 
THEN,  A  PRACTICE  ITEM  WILL  BE  PRESENTED 


TO  CONTINUE,  CLICK  ON  THIS  BOX - 


Presentation  Window 


<4, 

Lift 

Rocoiror 

<£> 
Listen  For 
Ton* 

1 

Switch-Book 

Bang  Up 

1 

ABC 

2 

DEF 

3 

GHI 

JKL 

UNO 

4 

5 

6 

pp 

1 

fwXYl 

1  9| 

* 

OPER 

0 

X? 

HERE  IS  THE  SIMULATED  KEYPAD. 


THE  SIMULATED  KEY  PAD 
HAS  16  CLICK  AREAS. 


TO  CONTINUE,  CLICK  ON  THIS  BOX 


B-4 


Presentation  Window 


Lift  Receiver-  Lifting  the  hand  held  part  of  the 
^  phone  from  its  cradle. 


lift 

Itccexver 


* 


P T*SS 

SvxtolvKookt 


~W 

Listen  For 
Tone 


Hast;  Up 


1 

ABC 

2 

OEF 

3 

MNO 

6 

GHI 

4 

JKL 

5 

PRS 

7 

TUY 

8 

WXY 

9 

-X- 

OPER 

0 

xt 

} 


Listen  For  Tone- Listening  for  a  dial  tone  or  short  busts  of 

tone  from  the  receiver.  A  dial  tone  indicates 
that  dialing  can  begin.  Busts  of  tone 
indicate  that  feature  activation  or 
cancellation  has  been  accepted. 

.  Press  Switch-Hook  -  To  Press  and  immediately  release  the 

two  buttons  under  the  receiver. 


Hang  Up  -  Retiming  the  receiver  to  Bs  cradle. 

Dial  Pad  -  The  1 0  pushbuttons  you  use  to  dial  a  number 
plus  the  "*■  and  pushbuttons. 

TO  CONTINUE.  CLICK  ON  "DIAL  PAD" 


Presentation  Window  i 


Presentation  window 


osq  ns  bust  mbilil  mi 

AGS  E8MHE  0?  A  P0A£TD(SE  HTHIFls 


TO  CONTINUE,  CLICK  ON  THIS  BOX 


Return 


Presentation  Window 


<4 

lift 

Receiver 


* 


Fths 
S*sfccl>-Hoo]d 


Listen  For 

,  Ton? 


Bug  Up 


1 

ABC 

2 

°¥ 

GHI 

4 

f 

MNO 

6 

PRS 

TUY 

8 

WXY 

9 

xt 

Your  assignment  will  be  to  read 
the  task  to  the  right  and  perform 
the  task  on  the  simulated  keypad 
on  the  left  For  example,  to  dial  a 
iber  click  on  ‘LIFT  RECEIVER’, 
th^click  on  ’LISTEN  FOR  TONE’, 

I  then  cU^k  on  the  extension  number 
(eg.  dick^n  '3'  then  '4'  then  '5'  and 
so  on).  Afwi^each  click 
appropriate  fefe^back  will  be 
provided  in  this 

PLEASE  CLICK  BELOW  TO 
CONTINUE... 


PLEASE  CALL  ^ 
^EXTENSION  3456 1} 


Return 


CLICK  HERE 
To  Continue 


None 
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PLEASE  CALL 
EXTENSION  34567 


Lift 

Receiver 


~w 

Listen  For 
-  Tone 


* 


Press 
Sestch-Hook! 


HssgUp 


1 

fl 

u 

m 

m 

m 

hi 

TUY 

8 

m 

□ 

This  is  a  dial  code  of  a 
or  *,  and  2  digits  which  can  be 
used  to  activate  or  cancel  a  feature 
such  as  CALL  FORWARDING,  CALL] 
PICKUP,  ABBREVIATED 
DIALING,  AND  CALL  HOLD. 


CLICK  HERE 
To  Continue 


7 


None 


Presentation  Window 


NEXT.  YOU  WILL  BE  ASKED  TO  DO  THE  SAM£L 
ITEM.  IF  THE  ITEM  DOES  NOT  MAKE  SENSfTO 
YOU.  PLEASE  CLICK  ON  THE  HELP<6n. 

PLEASE  CLICK  ON  THE  HELP  ICON  NOW. 


GOOD...  YOU  HAVE  SUCCESSFULLY  ASKED  FOR  HELP.  IF,  AFTER 
YOU  CLICK  ON  "HELP",  THE  SAMPLE  ITEM  STILL  DOES  NOT 
MAKE  SENSE  TO  YOU,  PLEASE  ASK  THE  INSTRUCTOR  OR 
RESEARCH  ASSISTANT  FOR  HELP. 


TO  CONTINUE,  CLICK  ON  THIS  BOX - 


Incorrect . The 

Hand  Points  To  The  Car  recti  a  Entered  Stew 


B-8 


You  have  correctly 
entered  the  steps  to 
activate  a  Call. 


1 

ABC 

2 

°IF 

GHI 

4 

JKL 

5 

MNO 

6 

PRS 

7 

TUY 

3 

WXY 

9 

■* 

OPER 

0 

xt 

Correctm  Entered  Stew 


DJLDtTir  BEEEOVEB 
2JL08TCE]  IMS  TTOJEE 
S^mMKSS  S4SS7 


CLICK  HERE 
To  Continue 


PLEASE  CALL 
EXTENSION  34567 


Program  Codes 
None 


msS 


-■■"■T  ===  Presentation  UJindoiii 

GOOD.... YOU  HAVE  SUCCESSFULLY  COMPLETED 
THE  SAMPLE  ITEM. 

NOW,  PLEASE  TFI  L  THE  RESEARCH  ASSISTANT 
OR  THE  INSTRUCTOR  THAT  YOU  HAVE  COMPLETED 
THE  INTRODUCTION.  HE  OR  SHE  WILL  LOAD  THE 
PRACTICE  ITEMS. 

AFTER  YOU  HAVE  COMPLETED  THE  PRACTICE 
ITEMS,  YOU  WILL  BE  GIVEN  A  PERFORMANCE  TEST. 

PLEASE.  IS)®  MOT  CLICK  THE  MUSE 
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APPENDIX  C 
POST-QUESTIONNAIRE 


C-0 


POST-QUESTIONNAIRE 


Last  Four  Digits  of  Social  Security  Number _  Date: _  Tlme:_ _ 

The  following  question  is  on  a  five  point  scale:  (Please  Circle) 

Tired  Alert 

1.  How  do  you  feel  right  now?  1  2  3  4  5 

2.  Please  place  a  check  mark  next  to  those  features  of  the  Navy  phone  system  that  you  would  now  feel  comfortable 
using: 

Abbreviated  Dialing _  Can  Forwarding _  Call  Pickup _  Call  Wailing _ 

Conference  Calling _  Transferring  a  Can _  Last  Number  Redial _  Call  Hold _ 

3.  In  your  own  words,  what  do  you  feel  is  the  purpose  of  the  “dial  tone’  when  using  the  Navy  phone 

system? _ _ _ _ _ 


4.  in  your  own  words,  what  do  you  feel  is  the  purpose  of  the  "switch-hook’  on  a  Navy  phone?. 


5 .  in  your  own  words,  what  do  you  feel  is  the  purpose  of  the  ’**  button  on  a  Navy  phone?. 


6.  In  your  own  words,  what  do  you  leel  Is  the  purpose  of  the  ***  button  on  a  Navy  phone? 


C-l 


Last  Four  Digits  of  Social  Security  Number. 


Date: 


Time: 


1. 


Do  you  think  the  Navy's  Consolidated  Area  Telephone  System  Is  difficult  to  use?. 
The  following  questions  are  on  a  five  point  scale:  (Please  Circle) 


2  Did  you  find  the  computerized  lesson: 


Easy.to.lise 

a  1  2 

Miserable 

b.  i  2 


Difficult  to  use 
4  5 

Enjoyable 

4  5 


3.  Were  you  frustrated  at  any  time  during  the  computerized  lesson?  if  so,  please  place  a 
check  mark  next  to  those  Navy's  phone  system  features  which  were  presented  in  a 
confusing  or  unsuitable  manner 


Abbreviated  Dlallnc 
Conference  Calling 


Call  Forwarding 
Transferring  a  Call 


Call  Pickup _  Call  Waiting 

Last  Number  Redial _ Call  Hole 


4.  Were  you  happy  with  the  feedback  provided  by  the  microcomputer  during  the  practice 
portion  of  the  lesson? _ If  not,  how  do  you  think  the  feedback  could  be  improved? 


5.  Would  you  recommend  computer-based  training  for  the  teaching  of  control  panel 

operation? _ 

6.  Any  other  comments? _ _ _ 
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